As a project, I was asked to build a customized feed using Yahoo Pipes. This service provides users the ability to remix data on the internet using an intuitive visual programming environment and has been on the internets for what will be three years come February 2010. At its inception, Tim O’Reilly lauded the service as a milestone in internet computing. Now that Pipes has had an opportunity to clean itself up and get comfortable, how has it really changed the landscape of internet computing?

The basis for Pipes is that anyone can use this tool to mix and combine the data on the web. To achieve this, Yahoo really dumbs down the interface for use by the lowest common denominator. The available modules have fixed input and output types and serve a very specific function. The unfortunate downside is that if a module does not exist which exactly fits your needs, you find yourself needing to build your idea around Pipes rather than the other way around. Several months after Pipes was released, Yahoo provided a Web Service module which sort of answers this shortcoming. You could POST your data in JSON format to whatever URL you desire (your own page, in this case) and it should respond with a properly formatted JSON or XML response. The challenge is that you should provide your own hosted solution if you want a “module” which handles the your feed data is just the right way. Yes, no professional web developer will be satisfied with a tool like Pipes and any complaining would be like being upset that your battery operated reciprocating saw doesn’t have enough power to cut down a tree in your backyard. Why should we be restricted to trimming our hedges when there are huge trees to fell? Bad analogies aside, the tool needs to fit the job, but an advanced module which uses pseudo-code or an existing simple programming language might be helpful in providing something useful for programmers while still serving the majority. I might even be willing to settle for a Branching or Conditionals module.

Shortcomings aside, the existing set of modules isn’t a bad set to have around as long as we’re still growing our opportunities, but the momentum on this product has severely backed off. With the exception of a half-hearted attempt in June at reassuring everyone that the Pipes team isn’t just sitting on their laurels, development for Pipes has significantly backed off in the past year. Yahoo has recently indicated their intention of hiring a software engineer for Pipes so maybe the current brain trust just needs a jumpstart.

As far as my own project, I initially wanted to create a widget for this blog. The widget would call the pipe I created with the URL of the displayed post, and the pipe would analyze the content, get important keywords, and search other like-minded sources for related articles and posts to the one being read. Yahoo Pipes provides a Term Extraction module which should have done the trick for the analyzation that was needed, but module mostly provided a vary narrow set of keywords which sometimes missed the entire point of my post. I also attempted to use a semantic tagging engine called Calais which I hoped would provide the same functionality. The problem I ran into was the way Calais would “tag” the specific page I was interested in. Calais depended on a description of the body being sent to the engine instead of it being able to do its own retrieval and scanning of the document. Rather than providing the keywords separately, it would modify the description which was sent with the keywords appended to the end of the description. This would be fine for human readers, but they made it very difficult to parse that information out of description for pragmatic use (especially with the Pipes toolset). If you’re interested in seeing this, you could load a copy of my pipe and play around with it. You’ll find Calais’ tags at the bottom of the description for each item in the debugger. I’m certain that some fancy regex work could have extracted the information I was looking for, but the more I played around with it the more frustrated I got so I let it be.

Ultimately, I ended up creating a “SuperSearch!” pipe which uses other people’s generated pipes to collect data from Twitter, Yahoo, Google, and other popular blogs. This is pretty close to what I was looking to create, the only exception is that the keyword extraction has been omitted. I envision being able to send the keywords from the tags of my post to this pipe and it would generate the “Releated Posts” link set that I want. I couldn’t finish this immediately as the WordPress widget system doesn’t let you use variables from the page that’s being viewed. Either way, the functionality is there, but it feels a lot like what everyone else is doing with Pipes.

I don’t think that Yahoo Pipes is useless. Pipes makes slightly annoying tasks much easier to complete, but as soon as you ask it to do some heavy lifting Pipes leaves me wanting a little more. I think it’s not what O’Reilly originally envisioned in a mashup creator. Not to mention Pipes lack of recent significant press coverage. News coverage had never blown up as big as when Pipes first launched and now most people are wondering if visual mashup editors are even relevant on the Internet with the recent termination of Microsoft Popfly and Google Mashup Editor. My guess is that developers are looking for something a little more robust and all encompassing.